We have described how, when using squared loss, the conditional expectation/probabilities provide the best approach to developing a decision rule.
In a binary case, the smallest true error we can achieve is determined by Bayes’ rule, which is a decision rule based on the true conditional probability:
\[ p(\mathbf{x}) = \mbox{Pr}(Y=1 \mid \mathbf{X}=\mathbf{x}) \]
- We have described several approaches to estimating \(p(\mathbf{x})\).